Storage optimization for large-scale distributed stream-processing systems

Toward Massive Query Optimization in Large-Scale Distributed Stream Systems

Existing distributed stream systems adopt a tightly-coupled communication paradigm and focus on fine-tuning of operator placements to achieve communication efficiency. This kind of approach is hard to scale (both to the nodes in the network and the users). In this paper, we propose a fundamentally different approach and present the design of a middleware for optimizing massive queries. Our appr...

Hybrid Regenerating Codes for Distributed Storage Systems

Distributed storage systems are mainly justified due to their ability to store data reliably over some unreliable nodes such that the system can have long term durability. Recently, regenerating codes are proposed to make a balance between the repair bandwidth and the storage capacity per node. This is achieved through using the notion of network coding approach. In this paper, a new variation ...

Scalable Planning for Distributed Stream Processing Systems

Recently the problem of automatic composition of workflows has been receiving increasing interest. Initial investigation has shown that designing a practical and scalable composition algorithm for this problem is hard. A very general computational model of a workflow (e.g., BPEL) can be Turingcomplete, which precludes fully automatic analysis of compositions. However, in many applications, work...

Distributed heterogeneous systems for large-scale graph processing

XFlow: Internet-Scale Distributed Stream Processing

Existing stream processing systems are designed for clustered deployments, and cannot adequately meet the scalability and adaptivity requirements of Internet-scale monitoring applications. Furthermore, these systems commonly optimize for a specific QoS metric, which may limit their applicability to diverse applications and environments. This paper presents XFlow, a generic distributed data coll...